Pages: 1 2 3 4 5 6 7 8 9 10 11
#Investigating memory leaks in Python
Memory leaks are difficult to trace down. A nice description of finding one is "A server memory leak", a story about tracing a memory leak in a Django application.
I'm trying to work out why a server process is slowly using more and more memory. The server parses a stream of text, and shoves stuff into a database. It's written in Python, and about the only external code it uses is SQLAlchemy
I found a combination of Meliae and objgraph useful. Pympler also seems useful (the tracker class particularly). Dowser also looks useful, and might have made things a bit easier..
...
Make a comment.
#A day on walk about (photos)
Click on an image for more details
...
Make a comment.
#Detecting, locating and fixing referenced based heap memory leaks (Java)
A common problem we all have to deal with is fixing memory leaks. In garbage collecting languages like Java we typically expect not to ever need to worry about memory management. However the limitations of how a garbage collector works means that we can still create leaks. Which defeats the advantage of a garbage collector.
How do leaks get formed?
The Java GC determines which objects to collect by looking for references to the object. If there is no references, then the object is no longer in use and can be safely destroyed without making the application unstable. If an object is now unused but still referenced, a leak occurs. Let's create a leak:
...
Make a comment.
#Strong, Soft, Weak and Phantom References (Java)
There are four distinct forms of references in the JVM, and indeed many of these apply to other garbage collected languages.
- Strong references
- Soft references
- Weak references
- Phantom references
It's important to know the differences, what affect they have on the collector and when you should be using them.
...
Make a comment.
#Rename all jpeg files by their exposure date (Bash)
I like to organise my files by when they were taken. So I wrote a script to do it for me. You will need the jhead tool installed. On Debian/Ubuntu/etc this is in the jhead package.
#!/bin/bash rm renameJPEG.restore.sh for file in *.jpg *.jpeg *.JPG *.JPEG; do if [ ! -e "$file" ]; then continue fi echo "Inspecting $file" DateTime=`jhead "$file" | grep "Date/Time"` FoundCount=`jhead "$file" | grep "Date/Time" | wc -l` if [ $FoundCount -gt 1 ]; then echo "Found too many results for: $file" continue elif [ $FoundCount -eq 0 ]; then echo "No valid headers found." continue else BaseFile=`echo $DateTime | awk '{ print $3 }' | tr ':' '-'` for i in `seq 1000`; do if [ ! -e "$BaseFile - $i.jpg" ]; then echo "New file: $BaseFile - $i.jpg" mv "$file" "$BaseFile - $i.jpg" echo "mv '$BaseFile - $i.jpg' '$file'" >> renameJPEG.restore.sh break fi done fi done
This will rename all the files to the format YY-MM-DD - i.jpg where i is a uniquifier. Note that if you've some files with bad headers then you will get the wrong output.
...
Make a comment.
#Deployment System Requirements
Over the past month, my colleague Kurtis and I have been engineering a fully automated deployment system for all of our projects at work. This system was created to remove the painful deployment process from our development environment. This article discusses the requirements that we had for our deployment system. I eventually hope to discuss the entire thing, how we built it, and what revisions we make.
Why Deploying Can Be Painful
Most programmers hate deploying code. I certainly did until this past year. Deploying code is (often times) much harder than writing the code itself. Deploying requires a great deal of knowledge: OS configuration and setup, cloud provider APIs or physical server installation quirks, numerous software components, system monitoring, access lists, and a lot of other things.
It's a pain.
...
Make a comment.
#NEVERFEAR.org on Github
NEVERFEAR.org is now on Github. We're slowly putting stuff up there in a central organisation. In the long term we hope to replace the horribly ugly projects page with our activities on Github.
We invite you to wander over to https://github.com/organizations/NEVERFEAR where you will find our main page.
Want to participate? If you want to join the team by all means comment on this post and we'll get back to you.
...
Make a comment.
#Web Designers vs Web Developers
It's been a long time. We deserve a post! Even if merely a link.
This is so true. Web Designers vs. Web Developers. Combat trousers are awesome. The principle reason I dislike suits is that there's never enough pockets. When you have more than 10 pockets in your trousers all of your development life, then you put on a suit and are restricted to 4. It the bane of my professional life.
...
Make a comment.
#What's Up With the 'File' Menu?
I'm no usability expert. In fact, I've been called tasteless by many people (my wife included), but what the hell is up with pretty much all programs having a 'File' menu?
http://dictionary.reference.com/ defines 'file' as:
...
Make a comment.
#Object pools (Python)
A very common problem in software engineering is optimising out the problem of construction and destruction of objects. I don't necessarily mean objects exclusively in the context of object orientated programming. I mean objects as in data objects, which could be a large array or a struct and so on.
If you don't need objects for a long time it does seem silly to construct an object, initialise it, then destroy it and discard it. Especially as memory allocation can be expensive.
The most common approach to minimising the impact of this is object pools. An object pool is a collection of pre-constructed objects, sometimes even pre-initialised objects. When a program wants to use an object for a short period of time they can check-out an object, configure it to their needs (if necessary), use the object and then check it back into the pool.
...
Make a comment.