Parse Command Line Arguments in Python

Most command line programs allow users to pass command line arguments into a program. Consider, for example, the unix ls:

ls -a

Python programs can utilize such techniques by using sys.argv and performing string operations on each element in the argv list. Here is an example script from Programming Python: Powerful Object-Oriented Programming with my own comments added to the script. (Of course, the Python standard library provides much more sophisticated getopt and optparse modules which should be used in production programs)

def getopts(argv):
    # Create a dictionary to store command line options
    opts = {}

    # Iterate through each command line argument
    while argv:
        # Look for the dash symbol
        if argv[0][0] == '-':

            # Create an entry in the opts dictionary
            opts[argv[0]] = argv[1]

            # Slice off the last entry and continue
            argv = argv[2:]

            # Otherwise just slice of the last entry since there
            # was no command line switch
            argv = argv[1:]

    # Return the dictionary of the command line options
    return opts

# Only run this code if this is a standalone script
if __name__ == '__main__':
    # Put the argv option into our namespace
    from sys import argv

    # call our getopts function
    myargs = getopts(argv)

    # Print our output
    if '-i' in myargs:


Detailed Explanation

Our goal in this script is to look for any command line switches that are passed to the script. To start with, we need to import the argv object from the sys module which is a list that provides the command line options passed to the script. On line 30, we call our getopts function and pass the argv object to the function. The program’s execution jumps up to line 3.

Line 3 creates an empty dictionary object that will store all command line switches with their arguments. For example, we may end up with ‘-n’:”Bob Belcher” in this dictionary. Starting on line 6, we enter into a while loop that examines each entry in argv.

Line 8 looks at the first character in the first entry of argv (remember that Python String implement the [] operator). We are just checking if the character in question is a dash (‘-‘) character. If this character is a dash, we are going to create an entry into our opts dictionary on line 11. Then we slice off the next 2 entries in argv from the beginning of the list (one for the switch, the other one for the argument).

Line 15 is the alternative case. If argv[0][0] is something other than a dash, we are going to ignore it. We just slice off the first entry in argv (only one this time since there is no switch). Once argv is empty, we can return the opts dictionary to the caller.

The program execution resumes on line 33. For demonstration purposes, it checks if the myargs dictionary has a ‘-i’ key. If it does, it will print the value associated with ‘-i’. Then line 36 prints out the myarg dictionary.

When run from the console, the script will show output such as this.

Patricks-MacBook-Pro:system stonesoup$ python -h "Bob Belcher" -w "Linda Belcher" -d1 "Tina Belcher" -s "Gene Belcher" -d2 "Louise Belcher" 
{'-h': 'Bob Belcher', '-w': 'Linda Belcher', '-d1': 'Tina Belcher', '-s': 'Gene Belcher', '-d2': 'Louise Belcher'}

Python Command Line Arguments

The sys module provides developers with an access point to command line arguments. Here is an example program that prints command line arguments to the console.

import sys

# The sys object has an argv field that is a
# list of the command line arguments passed to the program
# The first entry is the name of the script

Here is the program’s output when run from the command line.

Patricks-MacBook-Pro:system stonesoup$ python

The sys.argv object is a list of all command line arguments supplied to the script when run as a python program. Generally speaking, the first entry in the list is the name of the script. For more information on Python, see Programming Python: Powerful Object-Oriented Programming

Python Page Through A File

Many operating systems have command line tools that allow a user to page through a file in chunks. As a demonstration of how to read text files in Python, I used an example from Programming Python: Powerful Object-Oriented Programming.


def more(text, numlines=15):
    # This splits the text into a list object based on line
    # endings
    lines = text.splitlines()

    # Now continue to loop until we are out of lines
    while lines:
        # Slice off numLines into chunk
        chunk = lines[:numlines]
        # Remove numLines from the beginning of lines
        lines = lines[numlines:]

        # Now loop through each line in chunk
        for line in chunk:
            # and then print a line
        # Now ask the user if we want to keep going
        if lines and input('More?') not in ['y', 'Y']:

if __name__ == '__main__':
    # Import sys so that we can read command line arguments
    import sys
    # Next, we are grabbing the first argument from the
    # command line, and passing it the open function
    # which returns a file object. Calling read on this
    # object will dump the contents of the file into a String
    # which gets passed to our more function above
    more(open(sys.argv[1]).read(), 10)

Detailed Explanation

The comments in the code above are mine and explain what is going on in the program. The program starts by testing if this script is getting called as a standalone program or if we are importing this code as a module.

Assuming this is a standalone program, we import the sys module so that we can examine the command line arguments. The second command line argument needs to be a text file or this program will crash. We pass the name of the file to the open function, which returns a file object. Calling read() on the file object dumps the entire contents of the file into a String.

At this point, we pass the string into our more() function. It starts out by splitting the string by lines, which returns a list object. We start to loop through this list object, which continues until the list is empty.

Inside of the while loop, we slice off numLines from lines and store then in chunk. Then we remove those lines from the lines list. The next step is to print out each line in chunk. Once that is complete, we test if we still have more lines to print and if we do, we ask the user if they want to keep going or exit.

Here is the program output when run on my screen.

Patricks-MacBook-Pro:System stonesoup$ python
def more(text, numlines=15):
    lines = text.splitlines()

    while lines:
        chunk = lines[:numlines]
        lines = lines[numlines:]

        for line in chunk:
        if lines and input('More?') not in ['y', 'Y']:

if __name__ == '__main__':
    import sys
    more(open(sys.argv[1]).read(), 10)

Python Current Working Directory

Many programs have a need to figure out the current working directory (CWD) at runtime. The Python os package has a getcwd() function that returns a program’s CWD. This is an example taken from Programming Python: Powerful Object-Oriented Programming


import os, sys

# This prints the current working directory
print('my os.getcwd =>', os.getcwd())

# This prints the system path
print('my sys.path =>', sys.path[:6])

input('Press any key to exit')


There isn’t much going on in this program. The first line imports the os and sys modules. The next line calls the print statement and passes the value returned from os.getcwd(). That will print the current working directy.

The next line prints the system paths, limited to 6 paths. Finally there is an input statement that causes the program to wait until the user presses a key to exit the program.


Here is the output when ran on my system.

my os.getcwd => /Users/stonesoup/IdeaProjects/ProgrammingPython/PP4E/System
my sys.path => ['/Users/stonesoup/IdeaProjects/ProgrammingPython/PP4E/System', '/Users/stonesoup/Library/Application Support/IntelliJIdea2017.2/python/helpers/pydev', '/Users/stonesoup/IdeaProjects/ProgrammingPython', '/Users/stonesoup/IdeaProjects/ProgrammingPython/PP4E', '/Users/stonesoup/IdeaProjects/ProgrammingPython/PP4E/System', '/Users/stonesoup/Library/Application Support/IntelliJIdea2017.2/python/helpers/pydev']
Press any key to exit

Walk a Filetree in Python

Python has a powerful os.walk function that let’s a script walk through a file system in an efficient fashion. In this example, taken from Programming Python: Powerful Object-Oriented Programming, we will walk a file tree that will remove any p-code files that are present in the file tree.


Here is the code, with my comments added.

import os, sys

# Do we only want to find files only?
findonly = False

# Either use the CWD or a directly specified by command line arguments
rootdir = os.getcwd() if len(sys.argv) == 1 else sys.argv[1]

# Keep track of the found and removed files
found = removed = 0

# Walk through the file tree
for (thisDirLevel, subsHere, filesHere) in os.walk(rootdir):

    # Go through each file in the directory
    for filename in filesHere:

        # Check if it ends with .pyc
        if filename.endswith('.pyc'):

            # Assemble the full file name
            fullname = os.path.join(thisDirLevel, filename)
            print('=>', fullname)

            # Attempt to remove the file if asked to do so
            if not findonly:
                    # Attempt to delete the file

                    # Increment the removed count
                    removed += 1
                    # Handle the error
                    type, inst = sys.exc_info()[:2]

                    # Report that this file can't be removed
                    print('*'*4, 'Failed:', filename, type, inst)
            found += 1

# Output the total number of files removed
print('Found', found, 'files removed:', removed)

Detailed Explanation

This script functions in a findonly or remove mode. So the first variable we create on line 4 is a flag that decides if we are only looking for p-code files or if we are finding and removing such files. Next we create a rootdir varaible that is either the current working directory or a directory supplied by a command line argument. We create two variables on line 10, found and removed, which track how many files we have found and removed.

We get into the meat of the program on line 13 when we enter into a loop that iterates over os.walk. The os.walk function takes a directory path to start at and then goes through every single subdirectory in that file tree. It’s the standard way to walk a file tree in python. The function returns a tuple that includes the directory the os.walk function is currently examining, the number of subdirectories, and the number of files.

We create a nested loop on line 16 so that we can look at each file in the directory individually. On line 19, we check if the file ends with the .pyc extension. If it does, we use os.path.join to assemble a full file path in a platform agnostic fashion and then print out the full file path to the console.

If we are deleting files, we use os.remove on line 29 to attempt to delete a file. It’s critical that we wrap this in a try block because we may not hvae permission to delete the file. If deleting the file is successful, we increment the removed count. If it fails, the program execution will jump to line 35 and we report the error. The loop ends on line 39 and then repeats.

When the program is finished, we report how many files we found and removed.

Spring Security @RolesAllowed JSR250 Kotlin

Although Spring Security provides means to secure the web tier using XML markup, it’s also critically important that developers also secure backend method to ensure that methods. This post demosntrates an application in which a developer forgot to secure a web form but luckily the backend code is secured and provides a safe guard against such an error.

Enabling JSR250

Spring Boot takes a declaritive approaching to enabling method security, but we also need to provide it with an authentication manager.

//The next annotation enabled @RolesAllowed annotation
@EnableGlobalMethodSecurity(jsr250Enabled = true)
//We need to extend GlobalMethodSecurityConfiguration and override the configure method
//This will allow us to secure methods
class MethodSecurityConfig : GlobalMethodSecurityConfiguration(){

    override fun configure(auth: AuthenticationManagerBuilder) {
        //In our case, we are going to use an in memory authentication

fun configureAuthentication(auth: AuthenticationManagerBuilder){
            .withUser("bob").password("bob").roles("ADMIN", "USER")
            .withUser("gene").password("gene").roles( "USER")

We create a class that extends GlobalMethodSecurityConfiguration. We turn the method security on by annotating this class with @EnableGlobalMethodSecurity. By default, Spring uses it’s own @Secured annotation so if we want to use the JSR standard, we need to pass true to the jsr250Enabled annotation. Then our MethodSecurityConfig class needs to override the configure method and add an authentication scheme.

Readers may be wondering what the difference is between @Secured and @RolesAllowed annotations. There doesn’t seem to be much as both annotations seem to do the same thing. There is the possibility that other software libraries may act on @RolesAllowed and if there is such as concern, then use @Secured.

Securing Methods

Once we have enabled method security, we only need to decorate our specific methods. Here is a service class used in the example application.

//This is our class that we are going to secure
class BurgerService(@Autowired val burgerRepository: BurgerRepository){

    fun init(){
        //Just popuplates the DB for the example application
        val burgers = listOf(
                BurgerOfTheDay(name = "New Bacon-ings"),
                BurgerOfTheDay(name = "Last of the Mo-Jicama Burger"),
                BurgerOfTheDay(name = "Little Swiss Bunshine Burger"),
                BurgerOfTheDay(name = "Itsy Bitsy Teeny Weenie Yellow Polka-Dot Zucchini Burger"))

    fun destory(){
        //Clean up the DB when done

    //Any user can add a new BurgerOfTheDay
    @RolesAllowed(value = *arrayOf("USER", "ADMIN"))
    fun saveBurger(burgerOfTheDay: BurgerOfTheDay) =

    //But only adminstrators get to delete burgers
    @RolesAllowed(value = "ADMIN")
    fun deleteBurger(id : Long) = burgerRepository.delete(id)

    //Any user gets to see our Burgers
    @RolesAllowed(value = *arrayOf("USER", "ADMIN"))
    fun allBurgers() = burgerRepository.findAll()

The @RolesAllows annotation takes an array of allowed roles. In our case, we are letting anyone with the USER role to add burgers, but only ADMIN users are allowed to delete burgers. If a user without the ADMIN role attempts to invoke deleteBurger, an AccessDeniedException is thrown.

Catching Security Violations

Kotlin has no concept of checked exceptions, but Java users should note that Spring’s security exceptions are all RuntimeExceptions. If we want to report a security violation back to the user, we need to catch our security exceptions. Here is an example Controller class that handles security violations.

class IndexController(
        @Autowired val logger : Logger,
        @Autowired val burgerService: BurgerService) {

    fun doGet(model : Model) : String {
        model.addAttribute("burgers", burgerService.allBurgers().toList())
        return "index"

    fun saveBurger(
            @RequestParam("burgerName") burgerName : String,
            model : Model) : String {
        try {
            model.addAttribute("burgers", burgerService.allBurgers().toList())
            model.addAttribute("info", "Burger has been added")
        } catch (e : Exception){
            when (e){
                is AccessDeniedException -> {
          "Security Exception")
                else -> logger.error(e.toString(), e)
        } finally {
            return "index"

    fun deleteBurgers(
            @RequestParam("ids") ids : LongArray,
                      model: Model) : String {

        var errorThrown = false

        ids.forEach {
            try {

                //If the user doesn't have permission to invoke a method,
                //we will get AccessDeniedException which we handle and notify the user of the error
            } catch (e : Exception){
                when (e) {
                    is AccessDeniedException -> {
                        model.addAttribute("error", "Only Bob gets to delete burgers!")
              "Security error")
                    else -> logger.error(e.toString(), e)
                errorThrown = true
        model.addAttribute("burgers", burgerService.allBurgers().toList())
            model.addAttribute("info", "Deleted burgers")
        return "index"

You’ll ntoice that the deleteBurgers method looks for AccessDeniedException (which is handled by Koltin’s powerful when block). In our case, we report an error that only Bob get’s to delete burgers.

Putting it all together

Here is a video of a sample web application that demonstrates this code in action.

The code for the example application is available at my GitHub page.

You can also learn more about Spring MVC by referring to the following posts.

Spring Boot Caching with Kotlin

It’s fairly common for applications to continually ask a datastore for the same information repeatedly. Requests to datastores consume application resources and thus have a performance cost even when the requested data is small. The Spring Platform provides a solution allows applications to store information in an in memory caching system that allows applications to check the cache for the required data prior to making a call to the database. This example shows how to use Spring Boot and Kotlin to cache files that we are storing in the database.

Database Entity

We are going to define a database entity that stores files in a database. Since retrieving such data can be an expesive call to the database, we are going to cache this entity.

data class PersistedFile(
        @field: Id @field: GeneratedValue var id : Long = 0,
        var fileName : String = "",
        var mime : String = "",
        @field : Lob var bytes : ByteArray? = null)

You will notice that this class has a ByteArray field that is stored as a LOB in the database. In theory, this could be as many bytes as the system allows so ideally we would store this in cache. Other good candidates are entity classes that have complex object graphs and may result in the ORM generated complex SQL to retreive the managed object.

Enable Caching

Spring Boot defines a CachingManager internally for the application. You are free to use your own, but you need to configure your Spring Boot environment first.


You need to have spring-boot-starter-cache in your pom.xml or other dependency manager.



You also need to tell the environment to turn on caching by using the @EnableCaching

@EnableCaching  //Spring Boot provides a CacheManager our of the box
                //but it only turns on when this annotation is present
class CachingTutorialApplication

Decorate the Caching Methods

At this point, we only need to decorate the methods we want the environment cache. This is done by decorating our methods with the @Cacheable annotation and then providing the annotation with the name of a cache. We can also optionally tell the cache manager what to use for the key. Here is the code for our service class followed by an explanation.

//We are going to use this class to handle caching of our PersistedFile object
//Normally, we would encapsulate our repository, but we are leaving it public to keep the code down
class PersistedFileService(@Autowired val persistedFileRepository: PersistedFileRepository){

    //This annotation will cause the cache to store a persistedFile in memory
    //so that the program doesn't have to hit the DB each time for the file.
    //This will result in faster page load times. Since we know that managed objects
    //have unique primary keys, we can just use the primary key for the cache key
    @Cacheable(cacheNames = arrayOf("persistedFile"), key="#id")
    fun findOne(id : Long) : PersistedFile = persistedFileRepository.findOne(id)

    //This annotation will cause the cache to store persistedFile ids
    //By storing the ids, we don't need to hit the DB to know if a file exists first
    @Cacheable(cacheNames = arrayOf("persistedIds"))
    fun exists(id: Long?): Boolean = persistedFileRepository.exists(id)

The first method, findOne, is used to look up a persistedFile object from the database. You will notice that we pass persistedFile as an argument to cacheNames and then use the primary key as the key for this item’s cache. We can use the PK because we know it’s a unique value so we can help make the cache more performant. However, keep in mind that the key is optional.

We can also avoid another call to the database by storing if items exist in the database in the cache. The first time exists() is called, the application will fire a count sql statement to the database. On subsequent calls, the cache will simply return true or false depending on what is stored in the cache.

Putting it all together

I put together a small web application that demonstates the caching working together. I turned on the show sql property in the file so that viewers can see when the application is making calls to the database. You will notice that the first time I retreive the persisted file, there is sql generated. However, on the second call to the same object, no sql is generated because the application isn’t making a call to the database.

You can get the complete code from my GitHub page at this link.

Here are some links to posts that are related to concepts used in Spring Boot that we used today.