Server Secrets

N.B. I wrote this six or seven years ago.

As your web business grows bigger and bigger, you will probably need to begin using other peoples' software. You might install a shopping cart, or use a gallery page builder... but sooner or later, you'll have to learn more about scripts and files.

Have you ever had a "permissions problem" with your unix/linux server? That's what I'm here to explain.

Well, no, that's not quite true. I'm here to explain several Server Secrets. The "permissions problem" is only one of those secrets. Unfortunately, this stuff is incredibly boring. We're not talking plain boring here... we're talking sadistically boring. But, if you survive the experience, you may never have to ask for help with your server again.

For our purposes, unix and linux mean the same thing. If you are on a unix-like server, this essay applies to you. FreeBSD, Red Hat, Slackware, Linux, Solaris, all refer to flavors of unix.

I am not here to teach you unix. I am here to help you survive unix as a webmaster.

File Ownership

Let's start with perhaps the trickiest concept of all - file ownership.

I'm sure you're too young to recall this, but back in the mid 20th century, there was something known as a typewriter. A typewriter typically had one owner - that is, any specific typewriter was only used by one person.

In the same way, your laptop or desktop computer is probably a single-user machine. You don't need to worry about working around or interfering with somebody else's files. You might worry about people snooping, but that's a different issue. Generally speaking, each person has his or her own personal computer.

There was, of course, a gap between everyone (or everyone's secretary) having a typewriter, and everyone having their own personal computer. We had computers, but they were shared. On a small to medium scale, we had departmental computers. Each person had their own account. Same computer, different accounts, just like your local bank. That's how we kept things separate.

That's when unix came to be. Unix evolved on medium-sized computers, in universities and research laboratories. Same computer, different accounts. Therefore, the idea of ownership became extremely important. Every file, every transaction is owned by somebody, and who that somebody is, makes a difference as to how that transaction is handled.

Again, think about how things happen at your local bank. Every dollar, every coin - and most certainly every ball point pen - is owned by somebody, and in every case it's completely clear who that somebody is. You have an account number; other people have an account number, and those numbers do not get mixed up. Keeping those account ownerships separate is a fundamental part of the system.

In the same way, different bank employees can do different things. At my bank, the lady at the information desk can handle non-cash transactions such as depositing my Google payout check. But only the teller can handle a wire transfer.

Unix is built around that same kind of environment. Even if you have a unix machine all to yourself, you still have to deal with the same concepts of file ownership. Perhaps you have a dedicated server; you are the only one on the whole server. But, the concept remains. You need to deal with file ownership.

Do you recall I mentioned that unix is based on departmental computing? I can think of no reason whatsoever for you to care about that fact. Even so, that fact does make a difference to you!

The files on my unix account are owned by me. That makes sense, right? The same concept applies to your server. When you upload your files, you specify your ftp user name. Your uploaded files are owned by that ftp user name. That is, your files are owned by you. When you deposit money into your own account at the bank, the money is owned by you.

So, on unix, there are a number of different user accounts. Each account has its own files - and thus different files are owned by different users.

However, on unix, a user is also part of a group. Back in the late 20th century, different departments wanted to keep their stuff away from the other departments, but share things among themselves. For example, in the following file list:

drwxr-xr-x 4 ewbarnard users 4096 Mar 30 19:24 .
drwx------ 5 ewbarnard users 4096 Mar 30 19:20 ..
drwx--x--x 2 ewbarnard users 4096 Mar 30 19:23 cgi-bin
-rw-r--r-- 1 ewbarnard users 8 Mar 30 19:21 .htaccess
-rw-r--r-- 1 ewbarnard users 747 Mar 30 19:21 index.html
-rw-r--r-- 1 ewbarnard users 32 Mar 30 19:23 main.html
drwxrwxrwx 2 ewbarnard users 4096 Mar 30 19:24 members

the files are owned by user ewbarnard, but the files are also part of group users. In unix there are three levels of access: user, group, and others. In the above example, the file owner may update index.html (ewbarnard has read and write permission), and everyone else (both members of the same group, and others) has read permission.

In the next section, you'll begin to realize why this is important!

File Permissions

Caveat. This does not come anywhere close to being a complete explanation of unix file permissions. We are only covering the basics you need to deal with your server!

Unix file permissions are Read, Write, and Execute, abbreviated r, w, and x. The lack of a certain type of permission is shown with a dash. So, rwx means read, write and execute permission, and rw- means read and write permission but not execute.

"Directory" means the same thing as "folder." However, with unix they're always called directories. How do you know if it's a directory or an ordinary file? On the file listing, there will be a d in the left margin just before the file permissions list. In the above example, the first three items are directories, and the last item is a directory. A dash means it's a plain file. Anything other than d or - means your ftp program might get confused.

Unix directory permissions look the same as file permissions, but they are not the same! Directories have r, w, and x permission just like files... but r, w, and x don't mean the same thing! Unfortunately, this means we need to look at r, w, and x one item at a time.

What does file read permission mean? Just what you think it should. It means that if you know where the file is, you have permission to read it. What does directory read permission mean? Again, pretty much what you would expect. It means you're allowed to scan the directory, to find out what files it contains, and anything else known about each file - when it was created, how big it is, what its permissions are, and so on. So far, so good.

What does file write permission mean? It means you can edit the file; it means you can append to the file; it means you can truncate the file. It does not mean that you can delete the file! Can you see why? To delete the file, is to remove its directory entry. The delete operation requires directory permission, not file permission. It's the same with renaming a file... renaming or moving a file requires write permission for the directories involved. Unix doesn't care if you can even read the file, so long as you have the right directory permission.

Why do you care? When you begin working with CGI scripts, the above becomes terribly, horribly, sadistically significant. But we'll explain that in a bit.

I pretty much just explained what directory write permission means. If you do not have directory write permission, you can not create a file in that directory. Even if you can edit the file, you still can't delete it!

What does file execute permission mean? It means that - in theory - the file can be treated as a self-contained unix program. It might be a "real" program like ls or cp, or it might be a text file such as a php or perl program. Without the necessary x permission, unix will refuse to recognize it. In the case of a CGI script, you'll see a 500 error.

Directory execute permission, however, means something entirely different. You don't "execute" a directory. That is, you don't attempt to run it as a computer program. What else can you do with a directory? We already have scanning and updating the directory covered - that's directory read and write permission. What's left?

If you want to scan a directory to see what's there, that's directory read permission. But what if you already know what you need? You want main.html; you don't need to go looking for it. You just want it. If that directory has execute permission, you can have it. If it doesn't, you can't. Read permission allows you to look around and be nosy; execute permission allows you to have the file you need.

Why in the world do you care about the difference? Because of how your server admin set up your server. I'll explain, but there's one more thing we need to cover first.

Three Digits

Each file or directory, then, has three permissions (rwx) each, for the file owner (user), group, and others. That's why a unix directory listing shows three sets of permissions: drwxrwxrwx for a directory, and -rwxrwxrwx for a plain file.

Unix is all about abbreviations. Vowels are never used when something unpronounceable will do. The first letter will often be used in lieu of the entire word. cp stands for copy, od stands for octal dump, yes stands for you're going to be sorry you asked.

So. If you can get a letter to stand for a word, why not have a digit stand for a series of letters each standing for a word or phrase? To a unix person, that's called efficiency, and therefore the very height of coolness.

One of the coolest things in unix, is file permissions. (I'm just giving the tip of the iceberg, you will recall... if you happened to have observed how directory set group id changes from s to S with an nfs mount, you'd begin to appreciate ultimate coolness. However, we'll leave total coolness for another day.)

Read permission is assigned a value of 4. Write permission is 2, and execute permission is 1. Thus rwx is 7, rw- is 6, r-x is 5, r-- is 4, and --x is 1. When you slap user, group, others' permissions together, you have truths such as rwxrwxrwx is 777, rw-r--r-- is 644, rw-rw-rw- is 666, rwxr-xr-x is 755, and rwx--x--x is 711.

But, what does this mean??? Finally, I can explain!

Apache Server

On your server, you are one of the users. This makes sense, right? You have an ftp login, and the files in your domain are (mostly) owned by yourself - that is, they're owned by that ftp login. On a server, the "group" concept really doesn't matter. Either everybody else is in your group, or nobody is. But it's unix, and that means the "group" part of the permission stuff is still there.

Meanwhile, the Apache server itself, is a different user. The server itself owns its own files - and does not own your files. In the choice of "user, group, others", the server falls in the "others" category. (With one important exception; I'll get to that exception in just a bit!)

When a surfer blows on in to look at your awesome web page with 300k of fast-loading graphics, what actually happens? That surfer doesn't actually log on to your server. Instead, that surfer's browser sends a request to the Apache software which is logged in to your server. More precisely, the Apache software resides on your server, and listens for requests.

When Apache sees that surfer's request, it figures out which of your files to grab, and feed to that surfer. Can Apache grab everything you've got? No, it can't. First, it's restricted by the server configuration files, and by your .htaccess files. Second, it's restricted by your unix file permissions. We'll ignore that first restriction for now, and remain focused on the second.

If the file permission is 644 (-rw-r--r--), that means that "others" can read the file but not overwrite it. Apache is one of those others. What about the directory? Every file is in some directory, and therefore directory permission always counts. If the directory permission is 711 (drwx--x--x), the server can grab the file. If the directory permission is 700 (drwx------), the server can't touch it.

CGI Scripts

Until now, everything works exactly as you'd expect. You create and upload your files; surfers surf them; all is well. Until, that is, you install a CGI script.

Again, you need to think of your server as a shared departmental computer. Do you want your colleagues to be able to trash your files any time they have a mind to? Of course not. What about the ones who are not your colleagues? Do you want to allow anyone who has an account on that computer, to edit your personal files? Probably not.

On today's servers, this might sound silly. Especially on a dedicated server, where the only person there, is you. On a shared server (i.e., a virtual host), security is tight. Most people are limited to ftp access, and they wouldn't know how to find your files if they wanted to.

What's important is the mind set. If you think like the departments did a generation ago, you'll see what I'm getting at. Do you want the world at large to be able to overwrite your files?

Yes, actually, you do!

You see, your CGI scripts fall in the category of "the world at large." It's not you running the script. Sure, you go to the URL, click the button to update, but it's still not you!

Do you see why this is so? Your browser (which is showing you the button to click) is not logged into your server. You are a surfer. Your browser sends a request over to the Apache server software residing on your server. It's Apache who figures out that you wish to update one of your own files.

From a unix standpoint, however, you are not Apache and Apache is not you. You are two different users on the same computer. You're not even part of the same departmental work group.

Can you see what this means? You're probably several steps ahead of me at this point... but don't worry, it will get tricky in just a moment!

Suppose your CGI script needs to update your index.html page. Your file has 644 permission:

-rw-r--r-- 1 ewbarnard users 747 Mar 30 19:21 index.html

That means you (the owner) can update the file, but the world at large (including the Apache server!) can not. You have a permissions problem. You need to change the file permissions to 666 (-rw-rw-rw-). Any file that your script needs to update, must have 666 permission.

Caveat. There is an important exception to the above rule, which I will explain a bit later.

What if your CGI script needs to create, delete, or move files? Create, delete, and move are directory operations. That means the directory must have 777 permission. If your directory has 755 permission (drwxr-xr-x), the script will fail to create the file. And, chances are, it will not tell you that it failed, or tell you why it failed.

Did you catch that? If the script does not have directory write permission, it won't even know that something went wrong! It will blunder merrily along, and you'll have no idea something is broken.

Would you believe that you now know more about how unix files work, than most script writers? Based on my own observations, it's true!

CGI-Created Files

You can create a file that the CGI script can't touch. If you want the script to be able to read but not write the file, that's 644 permission. You own the file, and have read/write permission - that's the 6. The script falls in the "others" category, and thus has read-only permission - that's the second 4 of the 644.

In the same way, the script can create a file that you can't touch. If the script owns the file, and the file has 644 permission, you can read but not write the file. It's a two-way street. You and the script are two different users on the same computer. You can each read the other guy's files, and can update your own.

Finally - finally - we can wallow in the delicious subtleties of unix, which will bite you good and hard. Even some of the server admins don't seem to understand this!

Once again, consider your colleague in some other department. You and she are using the same computer. You need her to drop a copy of one of her files into your directory. Therefore, you give her an open directory like this:

drwxrwxrwx 2 ewbarnard users 4096 Mar 31 12:40 tmp

777 directory permission means that she can create a file in your directory, or copy a file into your directory, which amounts to the same thing. After she's dropped the file into your tmp directory for you, your directory might look like this:

drwxrwxrwx 2 ewbarnard users 4096 Mar 31 12:42 .
drwxr-xr-x 5 ewbarnard users 4096 Mar 31 12:40 ..
-rw-r--r-- 1 gopher squid 8 Mar 31 12:42 info.html

(In unix, . refers to the current directory, and .. refers to one directory up - that is, the current directory's parent directory.)

Do you see the problem we have here? gopher was kind enough to provide me the file info.html. I can read the file, but I can't update it! The file has 644 permission, which would be just fine if I owned the file.

Do you see that I can rename, copy, or delete the file? Rename, copy, and delete require directory write permission, and I have plenty of that. Well, copy also requires that I read the file - but I can do that. I just can't edit it!

So... what can I do? I can't change the file's permissions, because I don't own the file. That's right! The file is in my directory. But I can't touch it (except to delete it), because I don't own it. Only the file owner (or the server admin) can change its permissions.

A generation ago, this was easy. Give gopher a call, and ask her to open up the file. No problem; it literally happened all the time. But what if the file owner is your CGI script? Who are you going to call?

Hmm... well, let's think about this. Do you actually care? Are you supposed to be editing that script's files? Probably not - in which case, just leave the file alone, and everything will be fine.

Interlude

Well. This was sadistically boring, as promised. Let me remind you of the important points, before we proceed to the main event. You'd be amazed at how many server admins don't really understand the stuff you're wading through here!

For a CGI script to find, create, delete, or move files, the script needs suitable directory permission. This usually means that if the script is doing stuff in the directory, the directory permission needs to be set to 777. If stuff is mysteriously failing to happen, make sure the directory is where the script thinks it is, and that it's set to 777.

For a CGI script to be treated as a script, the script itself needs execute permission, i.e., 755.

If a file needs to be updated (e.g., a data file), the file must be owned by the script, or have 666 permission. If a file needs to be created, it's the directory which must have 777 permission. If the script creates the file, the script owns the file, so the actual file permission won't be an issue (until we hit the main event!).

If you need to make all of your files and directories script-accessible, log into your server via telnet or ssh, and execute the following command:

find . -exec chmod ugo+rw {} \\;

That command will take care of the current directory and all subdirectories.

The Main Event

Finally! The Event has arrived! This is where things get messy, and this is where I get the calls for help.

What happens when you change servers, or restore all of your files from a backup copy? Perhaps your webspace provider put you on a different server, or perhaps you moved the whole shootin' match to a different webspace provider. Perhaps the new server admin took care of the move for you. He took care of the search and replace, to get the new pathnames correct.

In every one of these cases, we have the same problem, and it's a messy one. Do you see the problem yet? I'll be quite pleasantly surprised if you do. Many server admins don't see the problem either!

This problem is not the admins' fault, as you'll see. The admins are merely caught in the middle. The problem is not really the programmer's fault either, by the way. The problem is just a natural outcome of how unix works.

Do you recall that file info.html owned by gopher? The file with 644 permission? Let's assume that Apache is actually user gopher. In other words, your CGI script owns the file. That file can be edited by your script, so we have no problem.

Now for the Main Event. Let's just suppose that somehow that file magically became owned by you rather than by the script. It still has 644 permission - but suddenly you can edit the file, and your CGI script can not. If the script needs to edit the file, can you see how this can be a problem? You've swapped horses in midstream. You've yanked the rug out from under it. You changed the rules in the middle of the game. Your script might not recognize what just happened - and therefore you'll have no clue that something is wrong.

Give the file 666 permission rather than 644 permission, and all is well. Do you see why this is so? With 666 permission, the script can edit the file regardless of who actually owns the file.

Do you see the problem? If you somehow magically take ownership of the script's files, the script is screwed. It's your script, so that means that you just screwed yourself!

So. How do you screw yourself in such a fashion? You change servers. It's that simple.

How you changed servers, or why you changed servers, doesn't matter in the least. Perhaps your server admin upgraded you to a brand-new dedicated server. Perhaps you changed ISPs. That does not matter - it's the move itself that killed your script. You will have the same problem, by the way, if you had to restore your files from a backup. Disasters happen; that's why we make backups; recovering from the disaster can kill your script.

How do you make a backup? You make a copy of all your files and put them somewhere. Right? If you can't read the file, you can't make a backup copy of that file. This makes sense, right?

What if the CGI script owns the file? As long as you can still read the file, you can make the backup. That's what's important. So far, so good.

What if we move servers, though? How exactly do you move servers? What you do is, make a backup copy of all your stuff. Then, you unload all that stuff onto your new server. You are restoring your stuff from a backup. Sure, it's a new server, but from a unix standpoint it's a plain ole file restore.

When you do the file restore, you're screwed. It's that simple! And I can guarantee you there are plenty of server admins out there who will happily move your stuff, not realizing that they're killing your script in the process.

What just screwed you? File ownership! When you restore the files to your account, the files become owned by you. Because they're suddenly yours, you're screwed.

Remember info.html? It has 644 permission - which means that only the owner can edit it. When it was owned by the script, all was well. But now that you own it thanks to the server move, the script is broken.

Okay, so you're screwed. How do you fix the problem? You can go in with ftp, and individually change the permissions on each directory, and each file, that's affected. Or, you can use the find instruction that I showed you above.

Caveat

Many servers run their CGI scripts precisely as I describe above. That is, you and your script are two entirely different persons, so far as file ownership is concerned.

But... what if your script pretended to be you? Wouldn't that simplify things? Of course it would. Never again do you have to worry about those subtleties of file ownership. What's yours is yours, and what's the script's is yours too.

But... how does the script become you? That's a very tricky thing to do in unix. What's important here, is for you to know when this is happening. Why is it important? Because it makes such a tremendous difference to how you set up your file permissions.

Okay, so you need to know if your scripts become you, or not. How do you find out? That's easy - you ask your server admin. What's difficult, is knowing what question to ask. There is no single correct question to ask! Like I said, this is a tricky issue in unix, and therefore truly cool.

Try asking the question this way: If I upload a file a file for my CGI script to edit, does the file need to have 644 permission, or 666 permission? Unfortunately, your admin might miss the point of the question. So, at the same time, ask this second question: If my CGI script creates a file, is that file owned by my ftp user id, or is it owned by the Apache server's user id? If your admin has no idea what you're asking, consider that a clue.

This bit of trickiness has a number of different names in unix; the most common are SuExec and CGIWrapper. All of these mean that your CGI script pretends to be you.

Remember that if you change servers - such as upgrading to a dedicated server - the answer to your question will probably change, and that means your file and directory permissions must change too!

Surely I have convinced you that using this trickiness would save a whole lot of hassle? Why do many unix server setups not use this trickiness? It's not a question of convenience, but one of security.

Server Security

It's time to back up, and think like your server admin. Do you want your server hacked? Of course not! Neither does your admin. Do you want some other idiot on your server bringing the thing to its knees, so that your domains become unavailable? Of course not! Neither does your admin.

Therefore, on a shared server, your admin will do various things to protect you from your colleagues, and to protect your colleagues from you. This is unix, and that means you're all sharing a departmental computer.

There are two basic approaches to CGI script security, and you already know what those are. The first approach says the script is not you, and the second approach says the script is you. If you understand that, you understand CGI security!

Not You

The first approach means you need to have your files and directories wide open. For your CGI script to create a file, that file's directory must have 777 permission. You can't get more open than that - there's nothing wider. To edit a file, the file must have 666 (or 777) permission. Again, that's as unsecure as it's possible to be in unix.

Why is this a good thing? Because you only need to make certain files and directories vulnerable. You can keep the rest of your files untouchable. Your CGI script therefore has very clear-cut boundaries. It cannot create new files or edit other files even if it wanted to. It can only touch the places where you have explicitly given it permission.

Using this approach, some of your stuff is wide open, but the rest of your stuff is untouchable. This is a standard (and secure) approach. Of course, you could do something really dumb like this:

find . -exec chmod ugo+rw {} \\;

What have you done? You've just made all of your files completely vulnerable. Your script can edit them, but so can anyone else who happens to be on your server. On a shared server, this is a problem. On a dedicated server, this is not a problem - until someone figures out how to hack your script.

Is You

The other approach is for the script to pretend to be you. This is the more common approach for a shared server. Do you see the primary reasons?

I'm sure you see the first reason... if you're on a shared server, you're likely to be less experienced. You'd probably rather not be bothered with the complexities of unix ownerships and permissions. Your CGI script pretends to be you, and everything works the way you would expect.

The other reason is more subtle. You're forced to be a good neighbor. Have you considered messing with your neighbor's files, either by accident or on purpose? Can you pop into the server via ftp, and mess with the other guy's files? Of course not - you don't own them. And now, neither can your CGI script, because the script is no more (and no less) privileged than you are.

Consider the other situation - that is, when the script does not pretend to be you. It does not pretend to be your neighbor either. If you both run CGI scripts, you both need to leave certain files and directories wide open. And, therefore, you are vulnerable to each other.

Therefore, rather than leave you vulnerable to your neighbors, your server admin will probably configure the Apache server in such a way that your scripts pretend to be you. Your neighbor's scripts pretend to be him, and therefore can't mess with your stuff.

On a dedicated server, you don't need to worry about malicious neighbors. You're the only user of the server. Therefore, the first approach makes more sense. The script does not pretend to be you, and therefore you can limit its access to only certain files and directories.

On a shared server, your greater danger is from your neighbors - remembering that your neighbor could be a victim of a hacking expedition. It makes more sense for your scripts to run as if they were you, sidestepping those permissions problems, and keeping you out of your neighbor's stuff.

Warning

Unix and arrogance go together. Larry Wall, the creater of Perl, calls it hubris, and declares it to be a mandatory trait. A generation ago, the standard answer to any unix question was, "read the man page." It may take you several days to figure out which man page to go read; that fact was both assumed and expected. No self respecting unix guru will waste his time with someone incapable of figuring out which man page to read; and all unix gurus are self respecting.

Don't be put off by the arrogance and condescension of this article. I was portraying the atmosphere endemic to all unix discussions. You do not, of course need to put up with such crap. Instead, just read the man page.