Playing with awk


Results 1 to 5 of 5

Thread: Playing with awk

  1. #1
    Join Date
    Jul 2002
    Location
    New Orleans, LA USA
    Posts
    986

    Playing with awk

    Most of the things I need to do now-a-days I have scripted and saved away for when I need them, or even on a cron schedule (read: I've become lazy with BASH scripting). So I started trying to solve some basic problems from scratch using filters and pipes to refresh myself. As an added bonus, I allowed myself only one line - so everything has to be pipped are otherwise segmented. At any rate, I managed to back myself into a corner without too much effort. I don't really need this function, but it is the fact that I can't readily make it happen that urks me...

    I started simple with stuff like let's see a unique list of everyone logged into this server. Fine, easy, done.

    Code:
    who | awk '{print $1}' | uniq
    Then I said ok, let's see what processes these users have open. Easy enough, mostly swap w for who.

    Code:
     w | awk '{print $1 " - " $8}' | uniq
    Moving on, I said to myself w doesn't catch things that are still running from a closed session (nohup, process &, etc.), so I'll use ps and /etc/passwd (Debian starts user accounts at PID 1000). Took me a little longer to build, but wasn't overly difficult.

    Code:
    awk -F: '{print $1,$3}' /etc/passwd | while read user pid; do (( pid >= 1000 )) && ps aux | grep $pid | awk '{print $1 " - " $11}' | uniq; done
    So far so good, except then I decided I want a summary style output instead of just every process for every user printed (for example, unique list of users with total number of processes next to each user name). That is when I hit a wall, because I now need to run two loops and store the result of the first while running the second - and I don't know how to accomplish that in one line.

    By picking a single user and not needing their id displayed, the code becomes trivial with something like:

    Code:
    ps -u 1000 | wc -l | awk '{print $1-4}'
    However, what if I want ps to cycle through all the users in /etc/passwd and return the user id and the total open processes for that id - to look like:

    Code:
    user1  20
    user2  15
    user3  11
    ...
    Any ideas? Only real restrictions I've placed on myself are stay in BASH and execute off one line.
    Last edited by trilarian; 08-27-2010 at 05:33 PM. Reason: grammar
    "Whenever you find yourself on the side of the majority, it's time to pause and reflect."

    -Mark Twain

  2. #2
    Join Date
    Sep 1999
    Location
    Cambridge, UK
    Posts
    509
    I assume you don't really want to get a list of users and running processes; the scenario is just something you made up to challenge yourself. That's a good way to learn. The actual result can be achieved in a much simpler way but since that isn't the point let's look at what you've got.

    You know how to get all the users and what they're running
    Code:
    awk -F: '{print $1,$3}' /etc/passwd | while read user pid; do (( pid >= 1000 )) && ps aux | grep $pid | awk '{print $1 " - " $11}' | uniq; done
    You also know how to count processes from one user. The original had awk printing $1-4. -1 discounts the header from ps but I'm not sure what the other 3 lines you're discarding are.
    Code:
    ps -u 1000 | wc -l | awk '{print $1-1}'
    Presumably, then, your next step would have been to try to combine them. Here I'll preserve the use of pid for UID and split lines for readability.
    Code:
    awk -F: '{print $1,$3}' /etc/passwd | while read user pid; do
     (( pid >= 1000 )) && ps -u $pid | wc -l | awk '{print $1-1}'
    done
    Then you will have noticed that you ended up with an unhelpful list of numbers
    Code:
    20
    15
    11
    I assume that the bit you are struggling with is mapping those numbers back to users.

    If you just want to see the results that's easily solved by echoing the username before running ps
    Code:
    awk -F: '{print $1,$3}' /etc/passwd | while read user pid; do
     (( pid >= 1000 )) && echo -n "$user " && ps -u $pid | wc -l | awk '{print $1-1}'
    done
    That will give you something like
    Code:
    user1 20
    user2 15
    user3 11
    Note the user of echo -n to print the username with no newline so the result of wc can follow on the same line, and the user of echo && to ensure that both the echo and the ps only run when the UID is 1000+.

    That's all very well but it relies on the fact that the output goes straight to the terminal for you to look at. You can't do anything with the numbers in your script. The way around that is to capture the ps output with a subshell.
    Code:
    awk -F: '{print $1,$3}' /etc/passwd | while read user pid; do
     (( pid >= 1000 )) && count=$(ps -u $pid | wc -l | awk '{print $1-1}') && echo "$user $count"
    done
    Is that what you're looking for?

  3. #3
    Join Date
    Jul 2002
    Location
    New Orleans, LA USA
    Posts
    986
    Quote Originally Posted by furrycat View Post
    I assume you don't really want to get a list of users and running processes; the scenario is just something you made up to challenge yourself.
    Aye, purely made up, classroom style to try and get me thinking creatively with awk in particular. That is part of the reasoning behind the one line deal, so I wouldn't move part of the code into BASH.

    Quote Originally Posted by furrycat View Post
    You also know how to count processes from one user. The original had awk printing $1-4. -1 discounts the header from ps but I'm not sure what the other 3 lines you're discarding are.
    I thought as much and was initially confused too, but I need it as such to be an accurate number. For example:

    Code:
    trilarian@Debian-Server:~$ ps -u 1000 | wc -l | awk '{print $1-4}'
    3
    trilarian@Debian-Server:~$ ps -u 1000 | wc -l | awk '{print $1-1}'
    6
    trilarian@Debian-Server:~$ ps -u 1000
      PID TTY          TIME CMD
     3705 ?        00:34:46 mythbackend
    25913 ?        00:00:00 sshd
    25914 pts/1    00:00:00 bash
    25996 pts/1    00:00:00 ps
    trilarian@Debian-Server:~$
    I concluded I was stripping ps, wc, and awk as my 3 extra lines (which normally would not be running except for when the script fires to check).

    The rest is what I was looking for. For some reason I was overlooking echo, /sigh. Oh well, good it is something simple than something critical with my reasoning process.

    Thanks for the help.
    "Whenever you find yourself on the side of the majority, it's time to pause and reflect."

    -Mark Twain

  4. #4
    Join Date
    Sep 2013
    Posts
    1

    Lightbulb using AWK for splitting flat file with variable column

    Hello, i have a flat file like that:

    C1L1;C2L1;C3L1;C4L1;C5L1;C6L1;C7L1;C8L1;C9L1;C10L1 ;C11L1;C12L1;
    C1L2;C2L2;C3L2;C4L2;C5L2;C6L2;C7L2;
    C1L3;C2L3;C3L3;C4L3;C5L3;C6L3;C7L3;C8L3;C9L3;
    C1L4;C2L4;C3L4;C4L4;C5L4;C6L4;C7L4;C8L4;C9L4;C10L4 ;C11L4;C12L4;
    C1L5;C2L5;C3L5;C4L5;C5L5;C6L5;C7L5;C8L5;
    C1L6;C2L6;C3L6;C4L6;C5L6;C6L6;C7L6;C8L6;C9L6;C10L6 ;C11L6;C12L6;



    I want to retrieve the first two columns and the the last two:

    I have this program AWK:


    awk -F";" '{
    for (j=1; j<2; j++)
    if ( j==1 )
    printf("%s",$j)
    if ( j!=1 )
    printf(";%s",$j)

    for (i=NF; i>NF-3; i--)
    printf("%s;",$i)
    print ""
    }' fichier_ioda_test

    And i have this results:

    C1L1;C2L1;C12L1;C11L1;
    C1L2;C2L2;C7L2;C6L2;
    C1L3;C2L3;C9L3;C8L3;
    C1L4;C2L4;C12L4;C11L4;
    C1L5;C2L5;C8L5;C7L5;
    C1L6;C2L6;C12L6;C11L6;

    it seems correct, but the problem is that the last two columns are reversed, SOMEBODY can help me please?
    Thanks
    Med

  5. #5
    Join Date
    Sep 2013
    Posts
    1
    D*n ông yêu d*n b* không ph?i vì nh?ng gì h? nói m* vì nh?ng gì h? bi?t l?ng nghe.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •